notebook.community

Edit and run 





In [1]:



    


import requests
from bs4 import BeautifulSoup
from robobrowser import RoboBrowser
import robobrowser
import csv

Basic Scraping

Actual webpage for CSO events has the search functionality in an iframe of this page: http://apps.mwrd.org/csoreports/CSO_Synopisis_Report

Looks relatively straightforward, but using some kind of Microsoft Ajax library so a direct post doesn't seem to work. Looks like will have to GET the page first, get a few parameters (including a cookie) that all seem to be contained in hidden inputs and post from that

Example Arguments
  • __EVENTTARGET:bttSearch
  • __EVENTARGUMENT:
  • __VIEWSTATE:"very long string"
  • __VIEWSTATEGENERATOR:229E5B6F
  • __EVENTVALIDATION: "long string"
  • txtStartDateSearch:8/17/2016
  • txtEndDateSearch:8/27/2016
  • ReportViewer1$ctl03$ctl00:
  • ReportViewer1$ctl03$ctl01:
  • ReportViewer1$ctl10:ss:
  • ReportViewer1$ctl11:standards
  • ReportViewer1$AsyncWait$HiddenCancelField:False
  • ReportViewer1$ToggleParam$store:
  • ReportViewer1$ToggleParam$collapse:false
  • ReportViewer1$ctl08$ClientClickedId:
  • ReportViewer1$ctl07$store:
  • ReportViewer1$ctl07$collapse:false
  • ReportViewer1$ctl09$VisibilityState$ctl00:None
  • ReportViewer1$ctl09$ScrollPosition:
  • ReportViewer1$ctl09$ReportControl$ctl02:
  • ReportViewer1$ctl09$ReportControl$ctl03:
  • ReportViewer1$ctl09$ReportControl$ctl04:FullPage




In [2]:



    


mwrd_url = "http://apps.mwrd.org/csoreports/CSO_Synopisis_Report"
mwrd_page = requests.get(mwrd_url)
print(mwrd_page.content)



    


















    






b'\r\n\r\n\r\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\r\n\r\n<html xmlns="http://www.w3.org/1999/xhtml">\r\n<head><title>\r\n\r\n</title>\r\n    <style type="text/css">\r\n\r\n        .auto-style6 {\r\n            width: 832px;\r\n            margin-right: 4px;\r\n        }\r\n        .style33\r\n        {\r\n            font-family: "Segoe UI";\r\n            font-size: medium;\r\n            color: #0B73B5;\r\n            font-weight: bold;\r\n            border-left-color: #9D9DA1;\r\n            border-right-color: #C0C0C0;\r\n            border-top-color: #9D9DA1;\r\n            border-bottom-color: #C0C0C0;\r\n            padding: 1px;\r\n        }\r\n        .style34\r\n        {\r\n            font-family: "Segoe UI";\r\n            font-weight: bold;\r\n            font-size: medium;\r\n            color: #0099FF;\r\n        }\r\n        \r\n        .auto-style7 {\r\n            width: 172px;\r\n            height: 12px;\r\n        }\r\n        .auto-style2 {\r\n            text-align: center;\r\n            height: 12px;\r\n            width: 266px;\r\n        }\r\n        .style28\r\n        {\r\n            text-align: center;\r\n            height: 12px;\r\n        }\r\n        .style45\r\n        {\r\n            width: 377px;\r\n            height: 12px;\r\n        }\r\n        .auto-style3 {\r\n            text-align: left;\r\n            height: 12px;\r\n            width: 266px;\r\n        }\r\n        .style55\r\n        {\r\n            text-align: left;\r\n            height: 12px;\r\n        }\r\n        .auto-style1 {\r\n            font-family: "Segoe UI";\r\n            color: #A0A0A0;\r\n            font-weight: bold;\r\n        }\r\n        .auto-style8 {\r\n            width: 172px;\r\n        }\r\n        .auto-style4 {\r\n            width: 266px;\r\n        }\r\n        .style56\r\n        {\r\n            width: 17px;\r\n        }\r\n        .style50\r\n        {\r\n            width: 134px;\r\n        }\r\n        .auto-style5 {\r\n            width: 287px;\r\n        }\r\n        .style51\r\n        {\r\n            width: 67px;\r\n        }\r\n        .style54\r\n        {\r\n            width: 377px;\r\n        }\r\n        </style>\r\n<link href="/CSOReports/WebResource.axd?d=-H-7euyzCq22q5_s0z2InJxtIytftx2o54bcs7ivORaSt7LZWcDwohM4Aq5xUot9qOGiHrRmOrQ5JsWCYQH_OzjIQeHjhkjs3q22cfm8lRxDtZvcwlT6qFqbngY9YLueWxk9ZzWG7A_hgjrWkoPbwg2&amp;t=635676347177046158" type="text/css" rel="stylesheet" /><link href="/CSOReports/WebResource.axd?d=hcXDZhXiidgsQUtEo5iT30XxEZrRWESl6IJ8_DnqsPZWlys1gJOXjT6uU5yeBtM6ZdM_UAic4mWU2eupLiGxF0jomZ5rR-YbEhOWM9SfifS1sQdOMl14YceJFopqctIIFok8plDZXxsA4T1xtkc3AOEfAOfk9JQTb_IWcymMfak1&amp;t=635676347177046158" type="text/css" rel="stylesheet" /></head>\r\n\r\n    <body>\r\n\r\n<form method="post" action="./CSO_Synopisis_Report" id="form1">\r\n<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="eXc4QggNokYHRtS7EeQUJFXdUHgCMwv2vF/YnnxYTTnMm2oFxT1300rAEzVqfrjja8Emj49zGh60vpM9Co/BZq4ujY4Do84wllRE0N6n9vFQP8D+P/3/k8vdiLTBONn5Ig+auHW8svQcYSP8e0g2AzWTn3xK4yeHq5WsSgtfzyK4nupWq2WHNJv6bsUwyJAIPWDSllEdf2n+Lo8jkXgp2OFnNQp5fTZAxm+EK2YlEzS5ZePoJx+mujfj2/HOxqPUClSGpRI4UUzJqbc8QLWJ9R+KnitYo0evsZIO6XL+y/XzIBU6NoMGNL8YL4ZNM8EyS2Pn6pkIpDQow9a9bWJKq5asGIRMYN8NS/V2i8yGps0sz62kPjH0A1Xaq1g3uAgaW9IoOHw8lrUT+wB4gwluQxO3sBYyx5udf1WBIvBjswtTS4+HnBBpeJ6DI2VhEJYtszfzZvVzxRs5FlA7sKr5WwJYIBg8vJA15AQClldTd6znrl7tuuqxcRYEi0Nav+nQ/FYdmeiXr+RzvuJoSuGOxjDvaWQMjdz4gV83jUfVN0Ew57zi5WRodpx8sCulzNZ9o2lMAaiTkJEMZ2rtfSSSchdk2OVE9d7Cxg2AK/+o6kQ8LfEeJkJ89VtQN5YtyWF89PRWc76anYsCwqvvCCMHtTV+ActXFE6fUvk9qKeEt9MexiAEUqVd6/vvi2gIRX3dwRLp9nk2KHVI+JBe+N0A72sz7AUYpc9PIbaJqL2tVAo3Sw0/Du8eaMegOZTeM9H5AdMKBKHYSR/IH5VtoidE7w3+lgwrQJs5eQ0r1UngS3gjo5wvhyXq0rrBfF6+kFjiVIEMr4EcaeGD0Z7hUw9x1Sfa7SjhBt0qImhYMhRshOMez1xan7ovFMoUMU0e2f+vIor6YskOxpHJUsh5T6yxBs8AO6HBC6qV0Kxw1dPqHMejfm09yDr/zEmxYMzSwaMgauEVGEvfhhpjsLFk3sWiDiloKKrNOFJapuAbiEIX8OOv2lOYKkfunBTy+GMNNNvAScpbfIeGCOhgWddj6/YzAXEsOau7g94qHw58Kl6s6XJoO/3l+g7tAX+lyUYG7rCbbkSaeZ+1jrK5APY2n9AEd4T01v8ORB2NZG3IAOR8drWUn9BMJJhsSBe1yrhAf7/bkoYEFlkbUdvkWydHtdTNfxieZSvv/orcKoveGFC/yH7mtzkYUm1iVvVsi1XQocN17FGlXNf01MgbKLAsEs/VXuo5+M5CrTWLeV3k3yZhyQ7IVc5fZe22q22vwJoKqefuH1fpXbbLBdZa56EuQtkThM4/OA/sfpQMYmT9C2gcRFMu/1w+Gnr8XtB75bEyrYmR7JWINarDAn4l6LNTuBjBFOLq/sP04pYS4NoJSnT/+24QTdbJnWZyk6hm4vDKafN/3ONYqnyoHDZijau4wfk9SQRhsXHn07jzHiDo1ca3xK4ap5Jj0DxvKt1WXH7cc8s33HGU0UC4MHrHBru7L2CeRwiZJt9+3OYRAUlwrifr0FmitbLvph7OpsCjUEC0ti+nonNtWRV/vk/jXbBEIzgkoQzxBxE10DqTFN26UH83LYeHuZYRPE3TvTNL+676HV+FszQKy5zEnDwOC4vN+ldadh59InZwGcYVvmiPp5LRLebuNKXOBQ8ReTYhb2iNVoC+21AEgz7vMDuHZpyVW1JlHSytgi4xk1aENeG00R/3STHY4l+yBVe2UwD6CCAdoZokoMf9qSNCinFZ2kEpS5LQwfUTpey9UoWk9oftW7L37KOvv0SjzeiZao60xhfGGU/F6bydtF2XKFdHP+n+AlLtpoybDA2uTsr4CpMwVyETSreVSsV9qnQ8EPbeZ9MFSR3dBLUP0pBUYKqh8Fsq5Xe1z2UmFs1dHQMlT86KOB8mLG0sDUqC1mFYOOageDhs8GJ58AMFyB7HCP0LEQL9pybP6OvlozDymilHKFz1G3CS/HWKxmcRCyTbBXzhsKIqbd9s3FBN4133Dt9MNv+bgu004pQz/K+6/hq/tJsZFZ6ORbp30mkR6/8uQzbDeIMgpr6/cKd2y/9Qx/rUhmdR3m710BsQSog8zXhyAoSt1w0zGAjYEYxSIdeSLqaHD9uz5Lza/3LHrDTKYimdhqKj+WZbTXbY/aZvge3mo12sCzZZ+g/R6slBVxHR1nv39RoC3UYIulu7ItAtMiIWgGWdqeT1bPaCeS7rT/wvfSPDhScz45aYAJjrbCVH+V6zhDbiBnDF+ln8P1dO85+M/CKbRPKwlGWYXZwJFUv/gihUZtdwoWP8CME4P7Fv+lWF4GCXvgZc9QjV80GXgC2/0jwfscU6EOnCWezSq68HQMpQkGQ5kv01PIMqCdWW6ll4ck+N5SPWQRvW6Ul+Men2m0osqr4zfRttincKcYlrotur4LNmvqxE/sseeW1sa338EXkcoYo8nOBSNvFcwEQH/osKJXyUvh4JTS7soraLJAhEj+8jEk8dfkh2+Li6CzVas2Rdw7lo/h9CpVUDUFdOt3ADbk7uGNLB4SNIP+0LAaHrFXyE5/OUA0tSmtoc1Bcy2GJS+JXpLjMEcconui4gB9UxNpb/QmcNKp1QHKzv6MZTHb/ZvDahovtVKM8q+K6b5EZ6L6d8cjuWY/p8/YucHy1hQ2ffRcw4in7RlY4hyldrHx3k/foXn3iFApsB4h81Dc9FedaroUfjovxUGaHHDuDoqtVAk/5C4xs1JpZM/W6dW604IEvjn4dR97+V9SjRqSV6YGn660Yik5LVYfZ4oK0GnYRCKTw0+9n/AvuTUdaPf0gz0HmtWwzLN6NIVv/4oFA5njB39hA2m9ezYBngT2P8G/OWLg==" />\r\n\r\n\r\n<script src="Scripts/WebForms/MsAjax/MicrosoftAjax.js" type="text/javascript"></script>\r\n<script src="Scripts/jquery-1.10.2.min.js" type="text/javascript"></script>\r\n<script src="/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource&amp;Version=11.0.2802.16&amp;Name=ViewerScript" type="text/javascript"></script>\r\n<script src="Scripts/WebForms/MsAjax/MicrosoftAjaxTimer.js" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=eE6V8nDbUVn0gtG6hKNX_G9QOacaAmfv6NR-YPHuyRQdnpFJ7f7IlxuJ_cP4ofAAauwmWXSGucV8OJlwmLKckVZVd9g-M5stk1ioyu-g4xNSdKqvcwLITr_YAELSOqGc0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=jwf4VSQi7LeShc44FJ-gAdLuc99HKQz5rjzA4v2nwjxXVNpqn2Hxe-T_BYt9yp_7VTokMtpF7-cuPs-ohpWaPyfs_HQFrWOeJkSimBLq_EDawNZm8M2maaI0NEA6Tvhd0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=mcAASOSteirJeuV-3by3UWaf51quEggIxxXsGkQBTIyXMZJpxTxaoCawJY9MPZ-at1iCsoD6b9dPKO3LkQMvAFMDF1LGEcby8xkVYSU1Vt01&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=uiErWUqGhD5_ddnGX4Xq_r_LmjGNtostwc9P0QwBEO2BMjuV-IPE6NrOJ1W7MaOSZ5vs-geGOoZabM7ndpmdoghGdtIgvEyRaW6Z05ijLPDXhRrjs-Fzxef-jI1f72I70&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=nai7mGcYyE2dix_TwveLDYY9vDp64LY07BMWVy-hU-YOLLyNEKiT4KtwgMTDXnVkAXlf7ginGf6LhEuqI5JCt5r0QOnC6-5j6L-RdG6eCZnPHvRN9bs3XCy5DyCYLrEg0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=LQ1q2RBSthBAZhliMjtG_0Q49RhZONJ8zCppqHNQBrecYL3S-eXYBixFuB07xqTnfO698g31neUM51mBSwH8beRLOP3a4sc9g8uQI3zfFolkU887bIkxv2GDwfgfXsoT0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=HEFpVKbnoeQjkjHkFKu3MAg-DznEPFF3MyNdTAH_2San7WCWAbpfyCtkByMH__wsGXdrhA7XAcBBHmWIfaHLPdnF45BiShpoOJLtCtY-_X_yYYTsywMCROARMQf2hiQV0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=VOYD8Xe7kQA0lniBYXVpAobNg56-nUjfzyGbMT6wA-tYJcH8QQeXGPAnnZqTq-vmIchsUJFNAYUW6WLcdGEDy9Im9BMHl31Yiw5RKqTI1MTGuWyumOKM0J2iC2L2FUes0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=24039AtbfhsNBYQDFUMYHILMzUWzp_rD8EfDltTZzCjnAoAk4i4VwHDPNvwHX38lXbtwj0B9UYUJCWtEGsRYgnsKZDGx6fyHGjhSlsTIorc1&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=myBlz7mQmGQE80F8-WeLsdIGknTXOfIceLSuDrRERQXZLYSRaCbMz4h2ifDO5FldhAz97_2UdeDQ6eb6QtXCp_clQjk0swjaPNSRJGxDiNmmoPzixWNPdZAsHjGF_aYK0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<script src="/CSOReports/ScriptResource.axd?d=NISXSzp87hD3qcCkP6NEPnUT_iMZMR1DIhLbFKBiL7yYFW6JI8A4sbXr1kCtMoKBSMp30wN742P-QVd9vsewNmL4_kJR-DMEX8KIKGwlYcQpbOQnnBi5t9-5Y-WS3_zv0&amp;t=fffffffff65665cf" type="text/javascript"></script>\r\n<input type="hidden" name="__VIEWSTATEGENERATOR" id="__VIEWSTATEGENERATOR" value="229E5B6F" />\r\n<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="oQBgOeCNuM2sNu/YL2hVwEjdndI/kY/2FO7tf3U1FmMHZQDyTnXqI1QYos1APcPtDBcvnZysmc2zfl7VZt5qZRaPMf8vYSBrI2iu1WKwOXLKO/W+Nam0NW2G7TNalLQuoIvBfbpk9XgmRkernjvIUQ7g8Y/kbJhCDakLXIQ/5fZVj1KUnTVKRtJiKUYV77hxlexslulVqeC7dO8PUZ5Gu66aqItLVXy1mhoHzRNCr3wEHwj46m/s5HRZtglXZ2cl6kP82wlEWyAae26yhl6K5K3Z2Z0uMNU2wUX6hUfa7C0a1FzJsk6v0XjSrM66UTP4dIM1zfxCOVQC7SNivgbZjJ0uFFfurteOfqDfLRqjjN1Xfc+9LYL+oE9NPRM9bGppuuOt9c+7dqYVuuLqogg6/dNARxyrIId/K6yKGRWor3j4NZg3l1zNqYXMOOH7jca6uSgETTKEZx4UkVCe3+PPco6WQtpslZsKTcaYE6krXDziNY/4fesloZiw/1ORhIPWPmKhOr8VmqOcn26+IPQ5I5BXANDQJ5Lr+79p745JKCnmvlmOL/lEtdXRi7GnQlPzz/enxLZGTf0lRMoKlpS2UO5I/Et8KG+0400tIFMYSYNpngb6Qbzm2aX8kH1inM0FudvdolTbbPPanoVdZc8Fav/L4KXYaFPMg9qg7wlg6928S5kRdRcniSphWx/2Ec/X0/gMyiV6S4iQ5bN0TkI2gGnG5M60DwsoAwNXBq2kZOpG20PpKIIBajLp96WUI2aFRSLI6XIbeJNMxZsQ3C85CMwvzZoYh542qWFM/uBcje+2mnOj8IyTjiLfWRBE+7+fuUHDtYIioh42zjqd+3tWNzW4aO5c92ozdGzIpvOTcsHhwXKg57bLnAqFDKpu06m4OjmV2TXhTeJLzr16dS+mCULQecyOyVQEs/GldurhuQooXIef/pHJtR5xoLZq4KCGQm1VdhnBKL9GnQm96DPXkhZXSaGyXlUxBt2ad/8I6ms0U1x0KCuevlIh2s9YD2gFR1lqITr15MJgpQKQth1fdIrjyrONYHjVJsq/Y5LNQcUT152VuneotTPlpQajlkVAjU6vSctuYSBdE5QXCR49tdHhftt/FbS2854tZJ19Yk4KlnsDXR0UFP6EDq/JTqV5yHITqGrxk0Jn1XlD27UzwsQZ3zPXQ5epJQfX0g28T99nm0yjXnVejyAw0H+qWHAD3xZyujg1Vwqn48sQHjYX/RG3S7Hp/91gkGTw6pfE6FP5B/Wo3KTo297URJAVp2MR4bIAeP+4Jw0CwAgnjGQXDHTRDfxddNNqealk3T2J8RnhOPICio3GfgvCFYF2LxAOWpeqek6jBSqcdKVJ8cx3m0ai+NkAPMelgXVW8N8OEsM0apQ+CKLWiCbK6qdW0/0tEKhlo/GiDrL+q3m96K0N1wmVzo85m2erefhiBqvzLO8BSfuNLqUHSPEvmqrtwlimDuxpZNm4gpZWlPRPhDBCt1zFIHyFC6kjlB2yYwLyFNM=" />\r\n    <table class="auto-style6">\r\n        <tr>\r\n            <td class="style31" colspan="6">&nbsp;<span class="style33">Search Criteria</span></td>\r\n        </tr>\r\n        <tr>\r\n            <td class="style31" colspan="6">\r\n                <img id="Image2" src="img/Horizontal_Line.jpg" height="16" width="797" />\r\n            </td>\r\n        </tr>\r\n        <tr>\r\n            <td class="style34" colspan="6">CSO Event Synopsis Report</td>\r\n        </tr>\r\n        <tr>\r\n            <td class="auto-style7">&nbsp;</td>\r\n            <td class="auto-style2">&nbsp;</td>\r\n            <td class="style28" colspan="2">&nbsp;</td>\r\n            <td class="style45" colspan="2">&nbsp;</td>\r\n        </tr>\r\n        <tr>\r\n            <td class="auto-style7">&nbsp;</td>\r\n            <td class="auto-style2">&nbsp;</td>\r\n            <td class="style28" colspan="2">&nbsp;</td>\r\n            <td class="style45" colspan="2">&nbsp;</td>\r\n        </tr>\r\n        <tr>\r\n            <td class="auto-style7">&nbsp;</td>\r\n            <td class="auto-style3">&nbsp;</td>\r\n            <td class="style55" colspan="2">\r\n                <img id="Image4" src="img/Calendar.gif" style="text-align: left" />\r\n                <span class="auto-style1">CSO Event Dates Range (MM/DD/YYYY)</span></td>\r\n            <td class="style45" colspan="2"></td>\r\n        </tr>\r\n        <tr>\r\n            <td class="auto-style8">\r\n                <select name="ddlCSODates" id="ddlCSODates" tabindex="1">\r\n\t<option selected="selected" value="0">--Choose a Date--</option>\r\n\t<option value="08/20/2016">08/20/2016</option>\r\n\t<option value="08/18/2016">08/18/2016</option>\r\n\t<option value="08/13/2016">08/13/2016</option>\r\n\t<option value="08/12/2016">08/12/2016</option>\r\n\t<option value="07/25/2016">07/25/2016</option>\r\n\t<option value="07/24/2016">07/24/2016</option>\r\n\t<option value="07/23/2016">07/23/2016</option>\r\n\t<option value="07/22/2016">07/22/2016</option>\r\n\t<option value="07/21/2016">07/21/2016</option>\r\n\t<option value="07/18/2016">07/18/2016</option>\r\n\t<option value="07/17/2016">07/17/2016</option>\r\n\t<option value="07/13/2016">07/13/2016</option>\r\n\t<option value="07/07/2016">07/07/2016</option>\r\n\t<option value="07/06/2016">07/06/2016</option>\r\n\t<option value="06/23/2016">06/23/2016</option>\r\n\t<option value="06/22/2016">06/22/2016</option>\r\n\t<option value="06/15/2016">06/15/2016</option>\r\n\t<option value="06/02/2016">06/02/2016</option>\r\n\t<option value="06/01/2016">06/01/2016</option>\r\n\t<option value="05/31/2016">05/31/2016</option>\r\n\t<option value="05/28/2016">05/28/2016</option>\r\n\t<option value="05/27/2016">05/27/2016</option>\r\n\t<option value="05/26/2016">05/26/2016</option>\r\n\t<option value="05/25/2016">05/25/2016</option>\r\n\t<option value="05/15/2016">05/15/2016</option>\r\n\t<option value="05/14/2016">05/14/2016</option>\r\n\t<option value="05/13/2016">05/13/2016</option>\r\n\t<option value="05/12/2016">05/12/2016</option>\r\n\t<option value="05/11/2016">05/11/2016</option>\r\n\t<option value="05/10/2016">05/10/2016</option>\r\n\t<option value="05/04/2016">05/04/2016</option>\r\n\t<option value="05/02/2016">05/02/2016</option>\r\n\t<option value="05/01/2016">05/01/2016</option>\r\n\t<option value="04/30/2016">04/30/2016</option>\r\n\t<option value="04/28/2016">04/28/2016</option>\r\n\t<option value="04/27/2016">04/27/2016</option>\r\n\t<option value="04/26/2016">04/26/2016</option>\r\n\t<option value="04/25/2016">04/25/2016</option>\r\n\t<option value="04/20/2016">04/20/2016</option>\r\n\t<option value="04/06/2016">04/06/2016</option>\r\n\t<option value="04/02/2016">04/02/2016</option>\r\n\t<option value="04/01/2016">04/01/2016</option>\r\n\r\n</select>\r\n            </td>\r\n            <td class="auto-style4">\r\n                <input type="button" name="bttSearchDay" value="Search" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;bttSearchDay&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, true))" id="bttSearchDay" tabindex="8" />\r\n                &nbsp;<input type="button" name="bttResetSearch2" value="Reset" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;bttResetSearch2&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, true))" id="bttResetSearch2" tabindex="9" />\r\n            </td>\r\n            <td class="style56">\r\n                <input name="txtStartDateSearch" type="text" id="txtStartDateSearch" tabindex="6" style="margin-left: 1px" />\r\n                \r\n            </td>\r\n            <td class="style50">\r\n                <input name="txtEndDateSearch" type="text" id="txtEndDateSearch" tabindex="7" style="margin-top: 0px" />\r\n                \r\n            </td>\r\n            <td class="auto-style5">\r\n                <input type="button" name="bttSearch" value="Search" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;bttSearch&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, true))" id="bttSearch" tabindex="8" />\r\n                &nbsp;\r\n                <input type="button" name="bttResetSearch" value="Reset" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;bttResetSearch&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, true))" id="bttResetSearch" tabindex="9" />\r\n            </td>\r\n            <td class="style51">&nbsp;</td>\r\n        </tr>\r\n        <tr>\r\n            <td class="auto-style8">&nbsp;</td>\r\n            <td class="auto-style4">&nbsp;</td>\r\n            <td class="style56">\r\n                &nbsp;\r\n            </td>\r\n            <td class="style50">\r\n                &nbsp;\r\n            </td>\r\n            <td class="style54" colspan="2">&nbsp;</td>\r\n        </tr>\r\n        <tr>\r\n            <td class="auto-style8">\r\n                \r\n            </td>\r\n            <td class="auto-style4">&nbsp;</td>\r\n            <td class="style56">&nbsp;</td>\r\n            <td class="style50">&nbsp;</td>\r\n            <td class="style54" colspan="2">\r\n                \r\n            </td>\r\n        </tr>\r\n    </table>\r\n    <br />\r\n    <br />\r\n    <br />\r\n    <div>\r\n    </div>\r\n    \r\n    <br />\r\n    <br />\r\n\r\n        <script type="text/javascript">\r\n\r\n           \r\n</script>\r\n\r\n        <noscript>\r\n\t&nbsp;Your browser does not support scripts or has been configured not to allow scripts.\r\n</noscript><span id="ReportViewer1_ReportViewer"><div id="ReportViewer1" onclick="if ($get(&#39;ReportViewer1_ctl04&#39;) != null &amp;&amp; $get(&#39;ReportViewer1_ctl04&#39;).control != null) $get(&#39;ReportViewer1_ctl04&#39;).control.HideActiveDropDown();" onactivate="if ($get(&#39;ReportViewer1_ctl04&#39;) != null &amp;&amp; $get(&#39;ReportViewer1_ctl04&#39;).control != null) $get(&#39;ReportViewer1_ctl04&#39;).control.HideActiveDropDown();">\r\n\t<div id="ReportViewer1_HttpHandlerMissingErrorMessage" style="padding:10px;display:none;overflow:auto;font-size:.85em;">\r\n\t\t<h2>\r\n\t\t\tReport Viewer Configuration Error\r\n\t\t</h2><p>The Report Viewer Web Control HTTP Handler has not been registered in the application&#39;s web.config file.  Add &lt;add verb=&quot;*&quot; path=&quot;Reserved.ReportViewerWebControl.axd&quot; type = &quot;Microsoft.Reporting.WebForms.HttpHandler, Microsoft.ReportViewer.WebForms, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91&quot; /&gt; to the system.web/httpHandlers section of the web.config file, or add &lt;add name=&quot;ReportViewerWebControlHandler&quot; preCondition=&quot;integratedMode&quot; verb=&quot;*&quot; path=&quot;Reserved.ReportViewerWebControl.axd&quot; type=&quot;Microsoft.Reporting.WebForms.HttpHandler, Microsoft.ReportViewer.WebForms, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91&quot; /&gt; to the system.webServer/handlers section for Internet Information Services 7 or later.</p>\r\n\t</div><span id="ReportViewer1_ctl03"><input type="hidden" name="ReportViewer1$ctl03$ctl00" id="ReportViewer1_ctl03_ctl00" /><input type="hidden" name="ReportViewer1$ctl03$ctl01" id="ReportViewer1_ctl03_ctl01" /></span><input type="hidden" name="ReportViewer1$ctl10" id="ReportViewer1_ctl10" /><input type="hidden" name="ReportViewer1$ctl11" id="ReportViewer1_ctl11" /><div id="ReportViewer1_AsyncWait" style="opacity:0.7;position:absolute;display:none;filter:alpha(opacity=70);">\r\n\r\n\t</div><div id="ReportViewer1_AsyncWait_Wait" style="display:none;position:absolute;">\r\n\t\t<table height="100%">\r\n\t\t\t<tr>\r\n\t\t\t\t<td width="32px" height="32px"><img src="/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource&amp;Version=11.0.2802.16&amp;Name=Microsoft.Reporting.WebForms.Icons.SpinningWheel.gif" alt="Loading..." height="32" width="32" /></td><td valign="middle" align="center"><span><font face="Verdana" size="4">Loading...</font></span><div>\r\n\t\t\t\t\t<a href="javascript:$get(&#39;ReportViewer1_AsyncWait&#39;).control._cancelCurrentPostback();" style="color:#3366CC;"><font face="Verdana" size="1">Cancel</font></a>\r\n\t\t\t\t</div></td>\r\n\t\t\t</tr>\r\n\t\t</table>\r\n\t</div><input type="hidden" name="ReportViewer1$AsyncWait$HiddenCancelField" id="ReportViewer1_AsyncWait_HiddenCancelField" value="False" /><table cellpadding="0" cellspacing="0" id="ReportViewer1_fixedTable">\r\n\t\t<tr id="ParametersRowReportViewer1" style="display:none;">\r\n\t\t\t<td colspan="3"></td>\r\n\t\t</tr><tr style="display:none;">\r\n\t\t\t<td colspan="3" align="center" bgcolor="#ECE9D8"><font size="1"><div id="ReportViewer1_ToggleParam">\r\n\t\t\t\t<input type="image" name="ReportViewer1$ToggleParam$img" id="ReportViewer1_ToggleParam_img" title="Show / Hide Parameters" src="/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource&amp;Version=11.0.2802.16&amp;Name=Microsoft.Reporting.WebForms.Icons.SplitterHorizCollapse.png" alt="Show / Hide Parameters" align="middle" onclick="void(0);WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ReportViewer1$ToggleParam$img&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false))" style="cursor:pointer;display:none;" /><input type="hidden" name="ReportViewer1$ToggleParam$store" id="ReportViewer1_ToggleParam_store" /><input type="hidden" name="ReportViewer1$ToggleParam$collapse" id="ReportViewer1_ToggleParam_collapse" value="false" />\r\n\t\t\t</div></font></td>\r\n\t\t</tr><tr style="display:none;">\r\n\r\n\t\t</tr><tr>\r\n\t\t\t<td valign="top" style="display:none;"><div>\r\n\t\t\t\t<span id="ReportViewer1_DocMap"><div id="ReportViewer1_ctl08" style="display:none;">\r\n\t\t\t\t\t<input type="hidden" name="ReportViewer1$ctl08$ClientClickedId" id="ReportViewer1_ctl08_ClientClickedId" />\r\n\t\t\t\t</div></span>\r\n\t\t\t</div></td><td width="4" height="100%" valign="middle" bgcolor="#ECE9D8" style="display:none;"><div id="ReportViewer1_ctl07">\r\n\t\t\t\t<input type="image" name="ReportViewer1$ctl07$img" id="ReportViewer1_ctl07_img" title="Show / Hide Document Map" src="/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource&amp;Version=11.0.2802.16&amp;Name=Microsoft.Reporting.WebForms.Icons.SplitterVertCollapse.png" alt="Show / Hide Document Map" align="top" onclick="void(0);WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ReportViewer1$ctl07$img&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, false))" style="cursor:pointer;" /><input type="hidden" name="ReportViewer1$ctl07$store" id="ReportViewer1_ctl07_store" /><input type="hidden" name="ReportViewer1$ctl07$collapse" id="ReportViewer1_ctl07_collapse" value="false" />\r\n\t\t\t</div></td><td valign="top"><div id="ReportViewer1_ctl09" style="position:relative;">\r\n\t\t\t\t<div id="VisibleReportContentReportViewer1_ctl09" style="display:none;">\r\n\r\n\t\t\t\t</div><div id="ReportViewer1_ctl09_ReportArea">\r\n\t\t\t\t\t<div NewContentType="Microsoft.Reporting.WebFormsClient.ReportAreaContent.None" ForNonReportContentArea="false" id="ReportViewer1_ctl09_VisibilityState" style="visibility:none;">\r\n\t\t\t\t\t\t<input type="hidden" name="ReportViewer1$ctl09$VisibilityState$ctl00" value="None" />\r\n\t\t\t\t\t</div><input type="hidden" name="ReportViewer1$ctl09$ScrollPosition" id="ReportViewer1_ctl09_ScrollPosition" /><span id="ReportViewer1_ctl09_Reserved_AsyncLoadTarget"></span><div id="ReportViewer1_ctl09_ReportControl" style="display:none;">\r\n\t\t\t\t\t\t<span></span><input type="hidden" name="ReportViewer1$ctl09$ReportControl$ctl02" /><input type="hidden" name="ReportViewer1$ctl09$ReportControl$ctl03" /><input type="hidden" name="ReportViewer1$ctl09$ReportControl$ctl04" id="ReportViewer1_ctl09_ReportControl_ctl04" value="FullPage" />\r\n\t\t\t\t\t</div><div id="ReportViewer1_ctl09_NonReportContent">\r\n\r\n\t\t\t\t\t</div>\r\n\t\t\t\t</div>\r\n\t\t\t</div></td>\r\n\t\t</tr>\r\n\t</table>\r\n</div></span>\r\n\r\n\r\n<script type="text/javascript">\r\n//<![CDATA[\r\nSys.Application.add_init(function() {\r\n    $create(Sys.Extended.UI.CalendarBehavior, {"format":"M/d/yyyy","id":"txtStartDateSearch_CalendarExtender"}, null, null, $get("txtStartDateSearch"));\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Sys.Extended.UI.CalendarBehavior, {"format":"M/d/yyyy","id":"txtEndDateSearch_CalendarExtender"}, null, null, $get("txtEndDateSearch"));\r\n});\r\nSys.Application.add_init(function() {\r\n    \r\nif (typeof Microsoft == \'undefined\' ||\r\n    typeof Microsoft.Reporting == \'undefined\' ||\r\n    typeof Microsoft.Reporting.WebFormsClient == \'undefined\' ||\r\n    typeof Microsoft.Reporting.WebFormsClient.ReportViewer == \'undefined\')\r\n    Sys.UI.DomElement.setVisible($get(\'ReportViewer1_HttpHandlerMissingErrorMessage\'), true);\r\n    $create(Microsoft.Reporting.WebFormsClient.ReportViewer, {"_internalViewerId":"ReportViewer1_ctl03","id":"ReportViewer1"}, null, null);\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Microsoft.Reporting.WebFormsClient._InternalReportViewer, {"ActionParamId":"ReportViewer1_ctl03_ctl01","ActionTypeId":"ReportViewer1_ctl03_ctl00","BaseHeight":"400px","BaseWidth":"1383px","BrowserModeId":"ReportViewer1_ctl11","DirectionCacheId":"ReportViewer1_ctl10","DocMapAreaId":"ReportViewer1_ctl08","DocMapHeaderOverflowDivId":"ReportViewer1_ctl08DocMapHeaderOverflowDiv","DocMapSplitterId":"ReportViewer1_ctl07","FixedTableId":"ReportViewer1_fixedTable","HasSizingRow":false,"PostBackToClientScript":function(){__doPostBack(\'ReportViewer1$ctl03\',\'\');},"PromptAreaRowId":"ParametersRowReportViewer1","PromptSplitterId":"ReportViewer1_ToggleParam","ReportAreaId":"ReportViewer1_ctl09","ReportViewerId":"ReportViewer1","UnableToLoadPrintMessage":"Unable to load client print control."}, null, null, $get("ReportViewer1_ctl03"));\r\n    $create(Microsoft.Reporting.WebFormsClient._SessionKeepAlive, {"KeepAliveBody":null,"KeepAliveIntervalSeconds":602,"KeepAliveUrl":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=SessionKeepAlive\\u0026ControlID=c6da6b0eb1fb4b5897f9cb40bb7ba343","id":"ReportViewer1_SessionKeepAlive"}, null, null);\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Microsoft.Reporting.WebFormsClient._AsyncWaitControl, {"ClientCanceledId":"ReportViewer1_AsyncWait_HiddenCancelField","DisplayDelay":1000,"FixedTableId":"ReportViewer1_fixedTable","ReportViewerId":"ReportViewer1","SkipTimer":true,"TriggerIds":["ReportViewer1"],"WaitControlId":"ReportViewer1_AsyncWait_Wait"}, null, null, $get("ReportViewer1_AsyncWait"));\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Microsoft.Reporting.WebFormsClient._Splitter, {"HoverColor":"#DDEEF7","ImageCollapse":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterHorizCollapse.png","ImageCollapseHover":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterHorizCollapseHover.png","ImageExpand":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterHorizExpand.png","ImageExpandHover":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterHorizExpandHover.png","ImageId":"ReportViewer1_ToggleParam_img","IsCollapsable":false,"NormalColor":"#ECE9D8","Resizable":false,"StoreCollapseField":"ReportViewer1_ToggleParam_collapse","StorePositionField":"ReportViewer1_ToggleParam_store","Vertical":false}, null, null, $get("ReportViewer1_ToggleParam"));\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Microsoft.Reporting.WebFormsClient._Splitter, {"HoverColor":"#DDEEF7","ImageCollapse":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterVertCollapse.png","ImageCollapseHover":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterVertCollapseHover.png","ImageExpand":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterVertExpand.png","ImageExpandHover":"/CSOReports/Reserved.ReportViewerWebControl.axd?OpType=Resource\\u0026Version=11.0.2802.16\\u0026Name=Microsoft.Reporting.WebForms.Icons.SplitterVertExpandHover.png","ImageId":"ReportViewer1_ctl07_img","IsCollapsable":true,"NormalColor":"#ECE9D8","Resizable":false,"StoreCollapseField":"ReportViewer1_ctl07_collapse","StorePositionField":"ReportViewer1_ctl07_store","Vertical":true}, null, null, $get("ReportViewer1_ctl07"));\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Microsoft.Reporting.WebFormsClient._ReportArea, {"NonReportContentId":"ReportViewer1_ctl09_NonReportContent","ReportAreaVisibilityStateId":"ReportViewer1_ctl09_VisibilityState","ReportControlId":"ReportViewer1_ctl09_ReportControl","ScrollPositionId":"ReportViewer1_ctl09_ScrollPosition","VisibleReportContentContainerId":"VisibleReportContentReportViewer1_ctl09"}, null, null, $get("ReportViewer1_ctl09"));\r\n});\r\nSys.Application.add_init(function() {\r\n    $create(Microsoft.Reporting.WebFormsClient._ReportPage, {"HiddenZoomLevelId":"ReportViewer1_ctl09_ReportControl_ctl04","StyleElementId":"ReportViewer1_ctl09_ReportControl_styles"}, null, null, $get("ReportViewer1_ctl09_ReportControl"));\r\n});\r\n//]]>\r\n</script>\r\n</form>\r\n          \r\n</body>\r\n</html>\r\n'

















In [3]:



    


mwrd_soup = BeautifulSoup(mwrd_page.content, 'html.parser')
input_vals = mwrd_soup.findAll('input')
for input_val in input_vals:
    print(input_val['name'])



    


















    






__VIEWSTATE
__VIEWSTATEGENERATOR
__EVENTVALIDATION
bttSearchDay
bttResetSearch2
txtStartDateSearch
txtEndDateSearch
bttSearch
bttResetSearch
ReportViewer1$ctl03$ctl00
ReportViewer1$ctl03$ctl01
ReportViewer1$ctl10
ReportViewer1$ctl11
ReportViewer1$AsyncWait$HiddenCancelField
ReportViewer1$ToggleParam$img
ReportViewer1$ToggleParam$store
ReportViewer1$ToggleParam$collapse
ReportViewer1$ctl08$ClientClickedId
ReportViewer1$ctl07$img
ReportViewer1$ctl07$store
ReportViewer1$ctl07$collapse
ReportViewer1$ctl09$VisibilityState$ctl00
ReportViewer1$ctl09$ScrollPosition
ReportViewer1$ctl09$ReportControl$ctl02
ReportViewer1$ctl09$ReportControl$ctl03
ReportViewer1$ctl09$ReportControl$ctl04

















In [4]:



    


s = requests.Session()
asp_headers = {
#     'Host': 'apps.mwrd.org',
#     'Origin': 'http://apps.mwrd.org',
#     'Referer': 'http://apps.mwrd.org/csoreports/CSO_Synopisis_Report',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
    'Connection': 'keep-alive',
    'Accept-Encoding': 'gzip, deflate'
}

mwrd_url = "http://apps.mwrd.org/csoreports/CSO_Synopisis_Report"
mwrd_page = s.get(mwrd_url)
mwrd_soup = BeautifulSoup(mwrd_page.content, 'html.parser')

all_inputs = mwrd_soup.find_all('input')

payload = {}

for inval in all_inputs:
    if inval.get('value'):
        payload[inval['name']] = inval['value']
    else:
        payload[inval['name']] = ''

payload['__EVENTTARGET'] = 'bttSearch'
payload['txtStartDateSearch'] = '08/13/2016'
payload['txtEndDateSearch'] = '08/27/2016'

payload['ReportViewer1$ctl11'] = 'standards'
#payload['ddlCSODates'] = '0'
#payload['ReportViewer1$ctl07$img'] = ''
# payload['__EVENTARGUMENT'] = ''
# payload['ReportViewer1$ctl09$VisibilityState$ctl00'] = 'ReportPage'

drop_keys = [
    'bttSearchDay',
    'bttResetSearch2',
    'bttSearch',
    'bttResetSearch',
    'ReportViewer1$ToggleParam$img'
]

for key in drop_keys:
    if payload.get(key):
        payload.pop(key)
# print('cookies: {} \n\n'.format(s.cookies))

for key in payload.keys():
    print(key)
# print(payload['ReportViewer1$ctl11'])
# print(payload['ReportViewer1$ctl09$VisibilityState$ctl00'])
mwrd_post_response = requests.post(mwrd_url, data=payload, headers=asp_headers, cookies=s.cookies)
mwrd_response_soup = BeautifulSoup(mwrd_post_response.content, 'html.parser')
# mwrd_response_soup = BeautifulSoup(resp.content, 'html.parser')
# print(mwrd_response_soup.find(id='ReportViewer1_ctl09'))

# Table with results has 9 cols, otherwise difficult to access
main_table_div = mwrd_response_soup.select("table[cols=9]")
# Get all table rows, other than the first which doesn't have the valign attribute, only searching in this subgroup
table_rows = main_table_div[0].select("tr[valign='top']")
# Create an empty list to add values to
table_list = []
# Go through each row, select the div in it, then grab the text from each div, add it to the row for the csv
for row in table_rows:
    row_cells = row.select('div')
    row_list = []
    # Get the text with the .text soup attribute, use .strip() to remove unnecessary whitespaces
    for cell in row_cells:
        row_list.append(cell.text.strip())
    table_list.append(row_list)

print(table_list)



    


















    






txtStartDateSearch
ReportViewer1$ctl09$ScrollPosition
ReportViewer1$ctl03$ctl00
txtEndDateSearch
ReportViewer1$ToggleParam$collapse
ReportViewer1$ctl03$ctl01
ReportViewer1$ToggleParam$store
ReportViewer1$ctl09$ReportControl$ctl04
ReportViewer1$ctl11
ReportViewer1$ToggleParam$img
ReportViewer1$ctl09$ReportControl$ctl03
ReportViewer1$ctl07$img
ReportViewer1$AsyncWait$HiddenCancelField
ReportViewer1$ctl07$store
__VIEWSTATEGENERATOR
__EVENTVALIDATION
ReportViewer1$ctl10
ReportViewer1$ctl09$ReportControl$ctl02
__EVENTTARGET
ReportViewer1$ctl07$collapse
ReportViewer1$ctl09$VisibilityState$ctl00
__VIEWSTATE
ReportViewer1$ctl08$ClientClickedId
[['Outfall Structure', 'Outfall Location', 'Tide Gate #', 'Waterway Reach', 'Plant', 'Open date/time', 'Close date/time', 'Gate Open Period'], ['DS-D34-AI', 'Westchester P.S.(N)', 'All', 'Addison Cr', 'Pumping Station Westchester', '8/20/2016 8:40:00 AM', '8/20/2016 11:30:00 AM', ':02:50:00'], ['DS-D34-AI', 'Westchester P.S.(N)', 'All', 'Addison Cr', 'Pumping Station Westchester', '8/18/2016 7:00:00 AM', '8/18/2016 8:05:00 AM', ':01:05:00'], ['DS-M27,DS-M28,& DS-M29 (RAPS)', 'Racine Ave P.S. (S)', 'All', 'SF SB Chicago R', 'Pumping Station Racine Avenue', '8/20/2016 9:30:00 AM', '8/20/2016 8:25:00 PM', ':10:55:00'], ['DS-M90 & DS-M91 (NBPS)', 'North Branch P.S. (E)', 'All', 'NBCR Lower (NSC Confluence)', 'Pumping Station North Branch', '8/20/2016 9:14:00 AM', '8/20/2016 3:00:00 PM', ':05:46:00'], ['DS-M90 & DS-M91 (NBPS)', 'North Branch P.S. (E)', 'All', 'NBCR Lower (NSC Confluence)', 'Pumping Station North Branch', '8/13/2016 12:32:00 AM', '8/13/2016 2:31:00 AM', ':01:59:00']]

















In [5]:



    


import csv 

with open('example_mwrd.csv', 'w') as mwrd_csv:
    mwrd_writer = csv.writer(mwrd_csv)
    for row in table_list:
        mwrd_writer.writerow(row)



    











In [6]:



    


# Create RoboBrowser, make sure to set parser to avoid errors later
browser = RoboBrowser(
    user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
    parser='html.parser'
)
# Open a session
browser.open('http://apps.mwrd.org/csoreports/CSO_Synopisis_Report')
# Identify the form by action, manually set important values
form = browser.get_form(action='./CSO_Synopisis_Report')
form['txtStartDateSearch'].value = '08/13/2016'
form['txtEndDateSearch'].value = '08/27/2016'
form['__EVENTTARGET'].value = 'bttSearch'
form['ReportViewer1$ctl11'].value = 'standards'
# form['ReportViewer1$ctl09$VisibilityState$ctl00'].value = 'ReportPage'

drop_keys = [
    'bttSearchDay',
    'bttResetSearch2',
    'bttSearch',
    'bttResetSearch',
    'ReportViewer1$ToggleParam$img'
]

# Remove extra keys
for key in drop_keys:
    form.fields.pop(key)

# Drop keys identified by printing all keys, then comparing to those posted in browser in Dev Tools
# for key in form.keys():
#     print('{}: {}'.format(key, form[key].value))

# Submit the form
browser.submit_form(form)

# Table with results has 9 cols, otherwise difficult to access
main_table_div = browser.select("table[cols=9]")
# Get all table rows, other than the first which doesn't have the valign attribute, only searching in this subgroup
table_rows = main_table_div[0].select("tr[valign='top']")
# Create an empty list to add values to
table_list = []
# Go through each row, select the div in it, then grab the text from each div, add it to the row for the csv
for row in table_rows:
    row_cells = row.select('div')
    row_list = []
    # Get the text with the .text soup attribute, use .strip() to remove unnecessary whitespaces
    for cell in row_cells:
        row_list.append(cell.text.strip())
    table_list.append(row_list)

print(table_list)



    


















    






[['Outfall Structure', 'Outfall Location', 'Tide Gate #', 'Waterway Reach', 'Plant', 'Open date/time', 'Close date/time', 'Gate Open Period'], ['DS-D34-AI', 'Westchester P.S.(N)', 'All', 'Addison Cr', 'Pumping Station Westchester', '8/20/2016 8:40:00 AM', '8/20/2016 11:30:00 AM', ':02:50:00'], ['DS-D34-AI', 'Westchester P.S.(N)', 'All', 'Addison Cr', 'Pumping Station Westchester', '8/18/2016 7:00:00 AM', '8/18/2016 8:05:00 AM', ':01:05:00'], ['DS-M27,DS-M28,& DS-M29 (RAPS)', 'Racine Ave P.S. (S)', 'All', 'SF SB Chicago R', 'Pumping Station Racine Avenue', '8/20/2016 9:30:00 AM', '8/20/2016 8:25:00 PM', ':10:55:00'], ['DS-M90 & DS-M91 (NBPS)', 'North Branch P.S. (E)', 'All', 'NBCR Lower (NSC Confluence)', 'Pumping Station North Branch', '8/20/2016 9:14:00 AM', '8/20/2016 3:00:00 PM', ':05:46:00'], ['DS-M90 & DS-M91 (NBPS)', 'North Branch P.S. (E)', 'All', 'NBCR Lower (NSC Confluence)', 'Pumping Station North Branch', '8/13/2016 12:32:00 AM', '8/13/2016 2:31:00 AM', ':01:59:00']]

Solution

ASP.NET form is partially AJAX, but not entirely. Needed to ensure that the form values submitted by whatever library we were using are the same as what is submitted by the browser. A few values were different, and some fields that did not need values were being submitted.

Potentially most relevant, the key ReportViewer1$ctl11 was blank on the first submit, but is standards in the successful post.

Helpful information on how this works was found in the Stack Overflow post here: http://stackoverflow.com/questions/27681731/python-requests-robobrowser-aspx-post-javascript

Many of the things that seemed mandatory were not, the most important factors were:

  • Sending with a current session cookie
  • Using ReportViewer1$ctl11
  • Setting the __EVENTTARGET to bttSearch
  • Dropping unnecessary inputs that were associated




In [ ]:

Content source: kbrose/rain

Similar notebooks:

notebook.community | gallery | about